WPI - CS - TR - 00 - 16 May 2000 Scalable Maintenance in Distributed Data

The maintenance of data warehouses is becoming an increasingly important topic due to the growing use, derivation and integration of digital information. Most previous work has dealt with one centralized data warehouse (DW) only. In this paper, we now focus on environments with multiple data warehouses that are possibly derived from other data warehouses. In such a large-scale environment, data updates from base sources may arrive in individual data warehouses in diierent orders, thus resulting in inconsistent data warehouse extents. We propose a registry-based solution strategy that addresses this problem by employing a registry agent responsible for establishing one unique order for the propagation of updates from the base sources to the data warehouses. With this solution, individual data warehouse managers can still maintain their respective extents autonomously and independently from each other, thus allowing them to apply any of the existing incremental maintenance algorithms from the literature. We demonstrate that this will indeed achieve consistency across all data warehouses. In order to achieve scalability of this approach, we further optimize this registry solution by partitioning the set of data warehouse managers into smaller data warehouse groups each equipped with their own registry. We present a partitioning algorithm for generating such a scalable DW group architecture. Finally, we analyze the performance of the proposed solutions based on a cost model, and demonstrate that the partitioned registry approach achieves substantially better performance than the registry solution.